APP08 Recovery (Real Time Clearing)

APP08 Purpose:

Clearing receives clearing records from APP01.

Clearing sends clearing records to SITE1 and sends clearing drop copy messages to APP02 for order sending firms that request them.

Use the following hyperlinks to jump to the desired section of APP08 documentation:

APP08_Recovery_Considerations

APP08_NTM_Control_Commands

APP08_Troubleshooting_Table

APP08_Monitoring_Considerations

APP08 Recovery Considerations:

Stopping/Restart Processes:

- Use NTM Control Utility - Service Control - Process Controller to stop/restart processes.

- APP08 process is closely tied to the OMD system. If moving APP08 to another node, see Application Recovery - OMD.docx

APP08 NTM Control Commands :

Resend Zero Cusip Messages:

- Use NTM Control Utility – Service Control – Realtime Clearing Options – Resend Zero Cusip Messages.

Send EOD Message:

- Use NTM Control Utility – Service Control - Realtime Clearing Options – Send EOD Message.

Set Outbound Sequence Number:

- Use NTM Control Utility – Service Control - Realtime Clearing Options – Set Outbound Sequence Number.

APP08 Troubleshooting Table:

APP08 Symptom	Impacts	Response
Node Crashes Evidenced by: - In Solarwinds (and outlook), node and processes will be reported down.	SITE1 will no longer be receiving clearing records. Order Sending firms requesting clearing drop copies will no longer be receiving them. Refer to: Server Recovery - Outbound Market Data.docx Server Specific Recoveries.	1) Refer to: Server Recovery - Outbound Market Data.docx Server Specific Recoveries. 2) Notify Management.

APP08 Monitoring Considerations:

Stats Monitors: APPGROUP01 to SITE1 App Queues Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate clearing trade delivery from APP01 processes to SITE1 fix engine. Monitor shows connection status between APPGROUP01 process and SITE1 fix engine, and processing statistics.	PROD MENU: Market Data Monitoring Menu To Exit: Close Window	- Color of data in columns - Status, - Write Queue	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APPGROUP01 processes are displayed as expected.	Not all APPGROUP01 services have been started or havn't processed any messages since monitor has been started. 1) Check status of services. 2) Check relevant service log files.
			Status is not CONNECTED.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process if other processes connecting to the same are showing similar issues; Otherwise, stop/restart source process. 2) Notify Production Support if issues.
			IPC Connected Queue size is non-zero value and not decreasing as expected.	Messages cannot be sent from source to destination unless IPC channel is connected. 1) Stop/Restart destination process so as not to accidentally delete queued messages; Do not stop/restart source process. 2) Notify Production Support if issues.

Stats Monitors: APPGROUP01 App Connect Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate clearing trade delivery from APP01 processes to SITE1 fix engine. Monitor shows processing statistics between APP01, APPGROUP01 and SITE1 processes.	PROD MENU: Market Data Monitoring Menu To Exit: Close Window	- Color of data in columns - InMsgs, - OutMsgs	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Status is Disconnected or Open	Firm is not connected. 1) Use NTM Control Utility RDC Service Controls to Open Channels. 2) Call Production Control if needed.
			InMsgs values are not equal to or greater than OutMsgs values. By design, APP01 sends all trades to APPGROUP01 process but APPGROUP01 only forwards non-test messages to SITE1 for actual clearing.	We may not be processing as expected. 1) Check APPGROUP01 log files to confirm inbound and outbound messages. 2) Work with Production Support if necessary.

Stats Monitors: APPGROUP01 IPC Instance Stats	To Start:	Key Indicators to Monitor:	Symptom:	Response:
Processes facilitate clearing trade delivery from APP01 processes to SITE1 fix engine. Monitor shows IPC channel processing statistics.	PROD MENU: Market Data Monitoring Menu To Exit: Close Window	- Color of data in columns - Service, - Hostname, - Msgs In, - Msgs Out	Data is RED.	Process is either down or multicast data is not being received by monitor. 1) Check status of process 2) If process is up, call Technical Services.
			Not all APPGROUP01 processes are displayed as expected.	Service has not been started. 1) Check status of service.
			Msgs In and/or Msgs Out are zero.	No messages have been sent/received since that monitor has been started. 1) Check APPGROUP01 log files. 2) Call Production Support if necessary.